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BACKGROUND OF THE INVENTIQM 

(a) Field of the Invention 

The present invention relates to a method and device for encoding 
and decoding stereoscopic video. More, specifically, the present invention 
relates to a method and device for encoding and decoding stereoscopic 
video into an encoding stream by using a conventional MPEG-4 MAC 
(multiple auxiliary component.) 

(b) Description of the Related Art 

A MVP (multi-view profile) method for extending two-dimensional 
video encoding techniques in the MPEG-2 technology has been 
developed as a conventional stereoscopic video encoding method. As to 
the MVP method, an encoding structure of a base layer for perfomiing 
encoding by using motion compensation corresponds to that of the 
MPEG.2 MP (main profile), and hence, one of the right and left images Is 
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reconstructed when the conventional 2-dimenslonal video decoder 
reconstructs the data of the base layer, thereby maintaining compatibility 
with the existing 2-dimensional video decoder system. An encoder of an 
enhancement layer uses correlation Infomiatlon provided between the 
right and left images to perform encoding. This is a method for performing 
encoding by using temporal scalability, and the MPEG-4 standard has 
also defined the temporal scalability using stereoscopic video encoding. 

The above-noted prior art has been disclosed by US Patent no. 
6,612,735 entitled "Digital 3D/stereoscoplc video compensation technique 
utilizing two disparity estimates." Regarding the '735 patent on the 
temporal scalability basis, a base layer uses a motion compensation 
algorithm and DOT (discrete cosine transform) base algorithm to encode 
Images of the left eye. and an enhancement layer uses disparity 
Information of between the base layer and the enhancement layer to 
encode the images of the right eye without motion compensation for the 
Images of the right eye. 

Also, as to the US patent no. 5,619.266 entitled "Digital 
3D/stereoscoplc video compensation technique utilizing disparity and 
motion compensated predictions," the base layer uses a motion 
compensation algorithm and DCT base algorithm to encode images of the 
left eye, and the enhancement layer uses information on the motion 
compensation between the Images of the right eye and disparity of 
between the base layer and the enhancement layer to encode the images 
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Of the right eye on the temporal scalability basis In the like manner of the 
'736 patent. This method achieves efficient compression rates by 
perfomning encoding by using motion and disparity Information, but the 
method has a complicated encoding structure and it is difficult to realize 
the method in the hardwired manner, and the method requires a large 
amount of calculation when processing HDTV Images. 

Therefore, in the case of encoding stereoscopic video by using 
temporal scalability, an additional multiplexer for transmitting respective 
encoding streams output by the base layer and the enhancement layer as 
a single stream is problematically needed in order to simplify a 
synchronization problem between the right and left images. 

Conventional multi-view video encoding methods include a 
method for perfomiing encoding by using a disparity map having disparity 
vector values of pixels, which has been disclosed by US Patent no. 
6,055,274 entitled "Method and apparatus for compressing multi-view 
video." 

The '274 patent encodes the total Image data of the first Image (a 
left image), and generates a disparity map having disparity vector values 
for respective pixels from the first Image and a second image (a right 
image) to perform encoding on the motion compensated disparity vectors, 
and uses a reconstructed first image after the encoding to disparity- 
compensate for a reconstructed disparity map and encode residual Image 
data of between the generated second image and the input second 



3 



wo 2004/059980 



PCT/KR2003/000132 



original image. This method problematically outputs a plurality of encoding 
streams, and additionally requires a multiplexer for transmitting them In a 
single stream format. 

In order to use the conventional MPEG codecs for 2-dlmensjonal 
images, and perfonn simple synchronization between the right and left 
images in the stereoscopic video, a method for reducing the respective 
right and left images by 1/2 and converting them Into 2-dimensional 
standard images has been proposed In five methods in the transactions of 
"3D video standards conversion, stereoscopic displays and applications," 
by Andrew Woods. Tom Docherty, and Rolf Koch (VII. California. Feb. 
1996, Proceedings of the SPIE vol. 1653A). 

The above technique is also published in US Patent no. 5,633,682 
entitled "Stereoscopic coding system." 

The Invention published as '682 selects odd-field Images for the 
images of the left eye and even-field Images for the images of the right 
eye to convert them into a single image, and accordingly performs MPEG 
encoding on the converted single Image with respect to the existing 2- 
dimenslonal images. This method considers a shuttering method for 
alternately displaying the right and left Images in the case of displaying the 
stereoscopic video, and the method is not suitable for polarized displays 
that concurrently display the right and left images. 



4 



wo 2004/059980 PCT/KR2003/000132 



10 



15 



20 



SUMMARY QF J^g iKiYCf.j-p-,r^^, 

It is an advantage of the present invention to maintain 
compatibility for using existing MPEG-4 encoding techniques and systems, 
and minimize complexity of synchronization between the right and left 
images. 

It is another advantage of the present invention to selectively 
control quality of images and encode the images according to Importance 
or complexity of the images, thereby improving encoding efficiencies. 

In the first aspect of the present invention, a method for encoding 
stereoscopic video including first and second images comprises: (a) 
encoding the first Image, and outputting a quantized video object and a 
motion vector of the first image; (b) receiving the first and second Images, 
and finding a pixel-based horizontal disparity map on the second image 
with reference to the first image; and (c) encoding the horizontal disparity 
map and outputting a quantized horizontal disparity map based on the 
pixel-based horizontal disparity map and a motion vector. 

In the second aspect of the present invention, a method for 
encoding stereoscopic video including first and second images comprises: 
(a) encoding the first Image, and outputting a quantized video object and a 
motion vector of the first image; (b) decoding the quantized video object 
output in (a), and reconstructing the first image; (c) receiving the first and 
second images, and finding a pixel-based horizontal disparity map on the 
second Image with reference to the first image; (d) encoding the horizontal 
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disparity map and outputting a quantized horizontal disparity map based 
on tlie pixel-based horizontal disparity map and the motion vector; (e) 
reconstructing the quantized horizontal disparity map output in (d), and 
outputting a reconstructed horizontal disparity map; (0 performing disparity 
5 compensation and outputting a pixel value of a disparity-compensated 
second image based on a pixel value of the first image reconstructed in 
(b) and a horizontal disparity vector value of the horizontal disparity map 
reconstructed in (e); and (g) perf^orming a residual process on the pixel 
value of the second image and the pixel value of the disparity- 

10 compensated second image output in (f) to output luminance residual 
texture, and encoding the luminance residual texture to output quantized 
luminance residual texture. 

In the third aspect of the present invention, a method for encoding 
stereoscopic video including first and second images comprises: (a) 

15 encoding the first image, and outputting a quantized video object and a 
motion vector of the first image; (b) decoding the quantized video object 
output in (a), and reconstructing the first image; (c) receiving the first and 
second images, and finding a pixel-based horizontal disparity map and a 
pixel-based vertical disparity map on the second image with reference to 

10 the first image; (d) encoding the horizontal disparity map and outputting a 
quantized horizontal disparity map based on the pixel-based horizontal 
disparity map and the motion vector; (e) encoding the vertical disparity 
map and outputting a quantized vertical disparity map based on the pixel- 
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based vertical disparity map and the motion vector; (f) reconstaicting the 
quantized horizontal disparity map output in (d), and outputting a 
reconstructed horizontal disparity map; (g) reconstructing the quantized 
vertical disparity map output in (d), and outputting a reconstructed vertical 
disparity map; (h) performing disparity compensation and outputting a 
pixel value of a disparity-compensated second image based on a pixel 
value of the first image reconstructed in (b), a horizontal disparity vector 
value of the horizontal disparity map reconstructed in (f), and a vertical 
disparity vector value of the vertical disparity map reconstructed in (h); and 
(i) performing a residual process on the pixel value of the second image 
and the pixel value of the disparity-compensated second image output in 
(h) to output luminance residual texture, and encoding the luminance 
residual texture to output quantized luminance residual texture. 

In the fourth aspect of the present invention, a method for 
decoding stereoscopic video including first and second images comprises: 
(a) receiving an encoding stream, and outputting quantized data of a video 
object of the first image, a motion vector, and quantized data of a 
horizontal disparity map; (b) decoding the video object and reconstructing 
the first Image based on the quantized data of the video object and the 
motion vector; (c) decoding the quantized data of the horizontal disparity 
map based on the quantized data of the horizontal disparity map and the 
motion vector; arid (d) perfomiing disparity compensation based oh the 
reconstructed first image and the decoded horizontal disparity map, and 
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reconstructing the second Image. 

In the fifth aspect of the present invention, a method for decoding 
stereoscopic video including first and second Images comprises: (a) 
receiving an encoding stream, and outputting quantized data of a video 
object of the first image, a motion vector, quantized data of a horizontal 
disparity map, and quantized data of luminance residual texture; (b) 
decoding the video object and reconstructing the first image based on the 
quantized data of the video object and the motion vector; (c) decoding the 
quantized data of the horizontal disparity map based on the quantized 
data of the horizontal disparity map and the motion vector; (d) decoding 
the quantized data of the luminance residual texture based on the 
quantized data of the luminance residual texture and the motion vecton 
(e) performing disparity compensation based on the reconstructed first 
image and the decoded horizontal disparity map, and outputting disparity- 
compensated luminance texture; and (0 adding the disparity-compensated 
luminance texture and the luminance residual texture reconstructed in (d) 
to reconstruct the second image. 

In the sixth aspect of the present invention, a method for 
decoding stereoscopic video including first and second images comprises: 
(a) receiving an encoding stream, and outputting quantized data of a video 
object of the first image, a motion vector, quantized data of a horizontal 
disparity map, quantized data of a vertical disparity map, and quantized 
data of luminance residual texture; (b) decoding the video object and 
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reconstructing the first image based on the quantized data of the video 
object and the motion vector; (c) decoding the quantized data of the 
horizontal disparity map based on the quantized data of the horizontal 
disparity map and the motion vector; (d) decoding the quantized data of 
the vertical disparity map based on the quantized data of the vertical 
disparity map and the motion vector; (e) decoding the quantized data of 
the luminance residual texture based on the quantized data of the 
luminance residual texture and the motion vector; <f) performing disparity 
compensation based on the reconstructed first image, the decoded 
horizontal disparity map, and the decoded vertical disparity map, and 
outputting disparity-compensated luminance texture; and (g) adding the 
disparity-compensated luminance texture and the luminance residual 
texture reconstructed in (e) to reconstruct the second image. 

In the seventh aspect of the present invention, an encoder for 
stereoscopic video including first and second images comprises: a video 
object encoder for encoding the first image, and outputting a quantized 
video object and a motion vector of the first image; a disparity estimator 
for receiving the first and second images, and finding a pixel-based 
horizontal disparity map on the second image with reference to the first 
image; and an auxiliary component encoder for encoding the horizontal 
disparity map and outputting a quantized horizontal disparity map based 
on the pixel-based horizontal disparity map output by the "disparity 
estimator and a motion vector output by the video object encoder. 
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In the eighth aspect of the present Invention, an encoder for 
stereoscopic video including first and second Images comprises: a video 
object encoder for encoding the first Image to output a quantized video 
object and a motion vector of the first Image, and encoding the quantized 
video object to output a reconstructed first image; a disparity estimator for 
receiving the first and second Images, and finding a pixel-based horizontal 
disparity map on the second image with reference to the first Image; a first 
auxiliary component encoder for encoding the horizontal disparity map to 
output a quantized horizontal disparity map. and decoding the output and 
quantized horizontal disparity map to output a reconstructed horizontal 
disparity map based on the pixel-based horizontal disparity map output by 
the disparity estimator and the motion vector output by the video object 
encoder; a disparity compensator for perfomiing disparity compensation 
and outputting a pixel value of a disparity-compensated second Image 
based on a pixel value of the reconstructed first Image output by the video 
object encoder and a horizontal disparity vector value of the reconstructed 
horizontal disparity map output by the first auxiliary component encoder; 
and a second auxiliary component encoder for perfomiing a residual 
process on the pixel value of the second Image and the pixel value of the 
disparity-compensated second Image output by the disparity compensator 
to output lunriinance residual texture, and encoding the luminance residual 
texture to output quantized luminance residual texture. 

In the ninth aspect of the present Invention, a decoder for 
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Stereoscopic video including first and second images comprises: a 
variable length decoder for receiving an encoding stream, and outputting 
quantized data of a video object of the first image, a motion vector, and 
quantized data of a horizontal disparity map; a video object decoder for 
decoding the video object and reconstructing the first image based on the 
quantized data of the video object and the motion vector output by the 
variable length decoder; an auxiliary component decoder for decoding the 
quantized data of the horizontal disparity map based on the quantized 
data of the horizontal disparity map and the motion vector output by the 
variable length decoder; and disparity compensator for perfomiing 
disparity compensation based on the reconstmcted first image output by 
the video object decoder and the decoded horizontal disparity map output 
by the auxiliary component decoder, and reconstructing the second image. 

In the tenth aspect of the present invention, a decoder for 
stereoscopic video Including first and second images comprises: a 
variable length decoder for receiving an encoding stream, and outputting 
quantized data of a video object of the first image, a motion vector, 
quantized data of a horizontal disparity map. and quantized data of 
luminance residual texture; a video object decoder for decoding the video 
object and reconstructing the first image based on the quantized data of 
the video object and the motion vector; a first auxiliary component 
decoder for decoding the quantized data of the horizontal disparity map 
based on the quantized data of the horizontal disparity map and the 
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motion vector; a second auxiliary component decoder for decoding the 
quantized data of the luminance residual texture based on the quantized 
data of the luminance residual texture and the motion vector; a disparity 
compensator for performing disparity compensation based on the 
reconstructed first image output by the video object decoder and the 
decoded horizontal disparity map output by the first auxiliary component 
decoder, and outputting disparity-compensated luminance texture and 
disparity-compensated chrominance texture; and a first adder for adding 
the disparity-compensated luminance texture output by the disparity 
compensator and the reconstructed luminance residual texture output by 
the second auxiliary component decoder. 



BRIEF DESCRIPTION OF T HE DRAWIMfSR 

The accompanying drawings, which are Incorporated in and 
constitute a part of the specification, illustrate an embodiment of the 
invention, and, together with the description, serve to explain the 
principles of the invention: 

FIG. 1 shows types and numbers of auxiliary components of 
vldeo_objectJayer_shape_extenslon defined by the MPEG-4 standards; 

FIG. 2 shows a stereoscopic video encoder according to a first 
preferred embodiment of the presfent invention; 

FIG. 3 shows a stereoscopic video decoder according to a first 
preferred embodiment of the present invention; 
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FIG. 4 shows types and numbers of auxiliary components of 
video_objectJayer_shape__extenslon which requires additional definition 
for encoding the stereoscopic video according to a preferred embodiment 
of the present invention; 

FIG. 5 shows types and numbers of auxiliary components of 
video_object_layer_shape_extension which requires additional definition 
when increasing the number of auxiliary components of the MAC (multiple 
auxiliary component) to greater than four; 

FIG. 6 shows a stereoscopic video encoder according to a 
second preferred embodiment of the present invention; 

FIG. 7 shows a stereoscopic video decoder according to a 
second preferred embodiment of the present invention; 

FIG. 8 shows an encoding stream output by the encoder 
according to a second prefen-ed embodiment of the present invention; 

FIG. 9 shows a stereoscopic video encoder according to a third 
preferred embodiment of the present invention; 

FIG. 10 shows a stereoscopic video decoder according to a third 
preferred embodiment of the present invention; 

FIG. 11 shows an encoding stream output by the encoder 
according to a third prefen-ed embodiment of the present invention; 

FIG. 1 2 shows a stereoscopic video encoder according to a fourth 
prefenred embodiment of the present invention; 

FIG. 13 shows a stereoscopic video decoder according to a fourth 
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preferred embodiment of the present invention; 

FIG. 14 shows an encoding stream output by the encoder 
according to a fourth preferred embodiment of the present invention; 

FIG. 15 shows a stereoscopic video encoder according to a fifth 
preferred embodiment of the present invention; 

FIG. 16 shows a stereoscopic video decoder according to a fifth 
preferred embodiment of the present invention; 

FIG. 17 shows an encoding stream output by the encoder 
according to a fifth preferred embodiment of the present invention; 

FIG. 18 shows a detailed block diagram for a video object 
encoder according to a prefen^d embodiment of the present Invention; 

FIG. 19 shows a detailed block diagram for an auxiliary 
component encoder according to a preferred embodiment of the present 
invention; 

FIG. 20 shows a detailed block diagram for a video object 
decoder according to a preferred embodiment of the present invention; 
and 

FIG. 21 shov\^ a detailed block diagram for an auxiliary 
component decoder according to a prefen^ed embodiment of the present 
invention. 



DETAILED DESCRIPTIQM OF THE PREFERRED FMRQDIMENTS 

In the following detailed description, only the preferred 
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embodiment of the invention has been shown and described, simply by 
way of illustration of the best mode contemplated by the inventor(s) of 
carrying out the invention. As will be realized, the invention is capable of 
modification in various obvious respects, all without departing from the 
Invention. Accordingly, the drawings and description are to be regarded as 
illustrative in nature, and not restrictive. 

The current MPEG-4 codec allocates auxiliary information 
including alpha, disparity, and depth which relate to respective video 
objects to an MAC (multiple auxiliary component) including three auxiliary 
components to encode them. 

The present invention relates to a method for using the MPEG-4 
MAC to encode the stereoscopic video, and auxiliary information to be 
allocated to the MAC will be defined. 

The current MPEG-4 MAC is added to MPEG-4 visual version 2 
so as to describe transparency of the video objects, and further, it defines 
auxiliary infomnation including disparity, depth, and additional texture 
relating to the video objects. 

FIG. 1 shows types and numbers of auxiliary components of 
video_objectJayer_shape_extension defined by the MPEG-4 standards. 

As shown, values of video_objectJayer_shape_extension in 
which the auxiliary components include the disparity type are defined as 
0001, 0010. 0011, and 0100. and no auxiliary components are defined to 
the values of from 1 101 to 1 1 1 1. 
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In the preferred embodiment, undefined auxiliary components of 
video_objectjayer_shape_extension values are defined to include 
needed auxiliary infomiation for more effective encoding of stereoscopic 
video, 

FIG. 2 shows a stereoscopic video encoder according to a first 
preferred embodiment of the present invention. The encoder uses 
auxiliary components defined by the existing MPEG-4 MAC shown in FIG. 
1 to encode the stereoscopic video data. 

As shown, the encoder comprises a disparity estimator 100, a 
video object encoder 200. an auxiliary component encoder 320, and a 
variable length encoder 600. 

The video object encoder 200 receives one image (a left image in 
the preferred embodiment) of the stereoscopic video, and outputs a 
quantized video object and a motion vector. The disparity estimator 100 
receives left and right Images, and finds a pixel-based horizontal disparity 
map of the right image with reference to the left image. That is, the 
disparity estimator 100 outputs position vectors moved to the horizontal 
axis as a horizontal disparity map so as to search and estimate to what 
positions of the left image the pixels of the right image are provided. 

The auxiliary component encoder 320 receives the horizontal 
disparity map which is an auxiliary component output by the disparity 
estimator 100 and the motion vector output by the video object encoder 
200, and outputs a quantized horizontal disparity map. 
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The variable length encoder 600 receives the quantized video 
object and the motion vector output by the video object encoder 200 and 
the quantized horizontal disparity map output by the auxiliary component 
encoder 320, performs variable length encoding, and outputs an encoding 
stream. 

FIG. 3 shows a decoder according to a first preferred embodiment 
of the present invention. 

As shown, the decoder comprises a variable length decoder 700, 
a video object decoder 800, an auxiliary component decoder 920, and a 
disparity compensator 1000. 

The variable length decoder 700 performs variable length 
decoding on the encoding stream output by the encoder of FIG. 2, and 
outputs quantized data of the video object, a motion vector, and quantized 
data of the horizontal disparity map. 

The video object decoder 800 receives the quantized data of the 
video object and the motion vector, and decodes the video object to 
decode the left image. 

The auxiliary component decoder 920 receives the quantized 
data of the horizontal disparity map and the motion vector, and decodes 
the horizontal disparity map. 

The disparity compensator 1000 receives a reconstructed left 
image and a reconstructed horizontal disparity map, performs disparity 
compensation based on a disparity vector of the horizontal disparity map, 
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and finally reconstructs a right Image. 

As described above, the disparity estimation is a process for 
searching and estimating to what positions of the left Image the pixels of 
the right image are provided, and position vectors moved to the horizontal 
axis are output as a horizontal disparity map, while position vectors moved 
to the vertical axis are output as a vertical disparity map. In general, a 
disparity map represents the horizontal disparity map because the right 
and left images of the ideal stereoscopic Image only have horizontal 
disparity values, and all the vertical disparity values are zero. 

The encoder and the decoder according to the first preferred 
embodiment are used for an ideal system which only has horizontal 
disparity values with the vertical disparity values of 0, and use the 
predefined MAC. 

That is, the encoder and the decoder according to the preferred 
embodiment use the value of vldeo_objectJayer_shape_extenslon with 
the conventional auxiliary components including the disparity type of FIG. 
1 to allocate the horizontal disparity map to the disparity type auxiliary 
component and encode the same. In this instance, the disparity map is 
found for the luminance data from among the luminance data and the 
chrominance data. 

Since the stereoscopic video by computer graphics are Ideally 
generated through a computer, the vertical disparity values can all be set 
to be zero so as to generate the stereoscopic video. Therefore, the 
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accurate disparity map can be found through the encoder and the decoder 
according to the first prefenBd embodiment, and the quality of the right 
Image is guaranteed to some degrees by this information. 

However, in the case of real Images captured by a Stereoscopic 
camera, the vertical disparity values are provided according to shapes and 
perfonnance of the actually manufactured stereoscopic camera differing • 
from the images by the computer graphics. In addition, since the accuracy 
of the disparity map by the horizontal disparity vectors found by calculation 
Is lowered below the accuracy of computer graphics images, the quality of 
the right Image reconstructed only through horizontal disparity Infonnation 
Is substantially worsened. Further, since the disparity map according to 
the first preferred embodiment has no infonnation on an occlusion area 
which is not provided to the left image but is to the right image, it is difficult 
to precisely reconstruct the right image. 

Accordingly, Information on the vertical disparity and residual 
texture as well as information on the horizontal disparity is added as 
information on the right image to the MPEG-4 MAC to define 
vldeo_objectJayer_shape_extension having the auxiliary components of 
FIGs. 4 and 5. 

The video_objectJayer_shape_extension of FIG. 4 includes 
values of from 1 101 to 1 11 1 which are not defined by the existing MPEG- 
4 MAC but are newly defined as an auxiliary component in the preferred 
embodiment of the present invention, and the 



19 



wo 2004/059980 



PCT/KR2003/000132 



video_objectJayer_shape_extenslon of FIG. 5 is newly defined In the 
preferred embodiment of the present invention so as to support four 
auxiliary components which are not supported by the MPEG-4 MAC. 

As shown In FIGs. 4 and 5, disparity infomiation is divided Into a 
horizontal disparity map and a vertical disparity map by the horizontal and 
vertical disparity values, the residual texture is classified as luminance 
residual texture and chrominance residual texture Infomiation, and 
auxiliary infomnation is selected according to importance and complexity of 
images to perform encoding. In this instance, the luminance residual 
texture infomation represents data of the left image reconstructed after 
decoding, the right image disparity-compensated by the reconstructed 
disparity map, and the residual image by the luminance component of the 
input right image. The chrominance residual texture infonriation represent 
data of the left image reconstructed after encoding, the right image 
disparity-compensated by the reconstructed disparity map, and the 
residual image by the chrominance component of the Input right image. 

FIGs. 6 and 7 respectively show an encoder and a decoder 
according to a second prefen-ed embodiment of the present invention. The 
encoder and the decoder establish the 

video_objectJayer_shape_extension which has two auxiliary components 
of the horizontal disparity type and the luminance residual texture type 
which are newly defined for encoding the stereoscopic video, and encode 
and decode the stereoscopic video. 
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As Shown in FIG. 6, the encoder comprises a disparity estimator 
100, a video object encoder 200, auxiliary component encoders 320 and 
340, a disparity compensator 400, an adder 500, and a variable length 
encoder 600. 

The video object encoder 200 receives one Image (a left Image in 
the preferred embodiment) of the stereoscopic video, outputs a quantized 
video object and a motion vector, and outputs a left image obtained by 
reconstructing the quantized video object. 

The disparity estimator 100 receives left and right images, and 
finds a pixel-based horizontal disparity map of the right image with 
reference to the left image. 

The auxiliary component encoder 320 receives the horizontal 
disparity map which is an auxiliary component output by the disparity 
estimator 100 and the motion vector output by the video object encoder 
200, generates a quantized horizontal disparity map, outputs the 
quantized horizontal disparity map, and outputs a horizontal disparity map 
obtained by reconstructing the quantized horizontal disparity map.. 

The disparity compensator 400 performs disparity compensation 
based on pixel values (luminance) of the reconstructed left Image output 
by the video object encoder 200 and the horizontal disparity vectors of the 
reconstructed horizontal map output by the auxiliary component encoder 
320, and outputs pixel values (luminance) of the compensated right image. 

The adder 500 performs a residual process on the pixel value 
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(luminance) of the right image and the pixel value (luminance) of the 
disparity-compensated right image output by the disparity compensator 
400 to output luminance residual texture, and the auxiliary component 
encoder 340 encodes the luminance residual texture to output quantized 
luminance residual texture. 

The variable length encoder 600 performs variable length 
encoding on the quantized video object and the motion vector output by 
the video object encoder 200, the quantized horizontal disparity map 
output by the auxiliary component encoder 320, and the quantized 
luminance residual texture output by the auxiliary component encoder 340, 
and outputs an encoding stream. 

FIG. 8 shows an encoding stream output by the encoder 
according to the second preferred embodiment of the present Invention. 
As shown, the encoding stream output by the encoder (a variable length 
encoder) includes macro blocks of the encoded video object (the left 
image), macro blocks of an auxiliary component AC[0] for the encoded 
horizontal disparity map, and macro blocks of an auxiliary component 
AC[1] for encoded luminance residual texture. 

FIG. 7 shows a decoder according to a second preferred 
embodiment of the present invention. 

As shown, the decoder comprises a variable length decoder 700, 
a video object decoder 800, auxiliary component decoders 920 and 940, a 
disparity compensator 1000, and an adder 1100. 
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The variable length decoder 700 performs variable length 
decoding on the encoding stream output by the encoder as shown In FIG. 
8, and outputs quantized data of the video object, a motion vector, 
quantized data of the horizontal disparity map, and quantized data of the 
luminance residual texture. 

The video object decoder 800 receives the quantized data of the 
video object and the motion vector, and decodes the video object to 
decode the left image. 

The auxiliary component decoder 920 receives th6 quantized 
data of the horizontal disparity map and the motion vector, and decodes 
the horizontal disparity map. 

The auxiliary component decoder 940 receives the quantized 
data of the luminance residual texture and the motion vector, and decodes 
the luminance residual texture. 

The disparity compensator 1000 receives a reconstructed left 
image and a reconstructed horizontal disparity map, and performs 
disparity compensation based on a disparity vector of the horizontal 
disparity map. 

The adder 1100 adds the disparity-compensated luminance 
texture from among the disparity-compensated data and the reconstructed 
luminance residual texture output by the auxiliary component decoder 940 
to reconstruct the right image. 

In the second preferred embodiment shown in FIGs. 6 and 7, the 
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left Image Is encoded and decoded as a single video object, and the right 
image is encoded and decoded using the MAC shown in FIG. 4. That is, 
the pixel-based horizontal disparity map found with reference to the left 
image is allocated to aux_comp_type[0], and the luminance residual 
texture which is residual image data of the luminance component is 
allocated to aux_comp_type[1l to perfomi encoding and decoding. 

The second preferred embodiment Is applicable to simple images 
which do not greatly influence deterioration of image quality when 
performing encoding without a vertical disparity map and chrominance 
residual texture information which is residual image data on the 
chrominance component, or to images which require no precise 
reconstruction. 

FIGs. 9 and 10 respectively show an encoder and a decoder 
according to a third preferred embodiment of the present invention. The 
encoder and the decoder establishes the 
video_obiectJayer_^shape_extension which has three auxiliary 
components of the horizontal disparity type, the luminance residual texture 
type, and the chrominance residual texture type which are additionally 
defined with reference to FIG. 4, and performs encoding and decoding. 

That is, the pixel-based horizontal disparity map found with 
reference to the left image is allocated to aux_comp_typeIO], the 
luminance residual texture is allocated to aux_comp_type[1], and the 
chrominance residual texture which is residual image data on the 
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chrominance component is allocated to aux_comp_typeI2]. The encoder 
and the decoder are usable for simple Images which do not greatly 
influence deterioration of image quality when performing encoding without 
a vertical disparity map. or for images which require no precise 
reconstruction. 

As shown in FIG. 9, the encoder according to the third preferred 
embodiment of the present invention comprises a disparity estimator 100, 
a video object encoder 200, auxiliary component encoders 320. 340, and 
360. a disparity compensator 400, an adder 500, and a variable length 
encoder 600. 

The components of FIG. 9 which perform the same or similar 
functions as those of the encoder of FIG. 6 have the same reference 
numerals, and no repeated descriptions on the components of FIG. 9 
which execute the same operations as those of the components of FIG. 6 
will be provided. 

The disparity compensator 400 performs disparity compensation 
based on pixel values (luminance and chrominance) of the reconstructed 
left image output by the video object encoder 200 and the horizontal 
disparity vectors of the reconstructed horizontal map output by the 
auxiliary component encoder 320, and outputs pixel values (luminance 
apd chrominance) of the compensated right Image. 

The adder 600 perfonns a residual process on the pixel values 
(luminance and chrominance) of the disparity-compensated right image 
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and the pixel values (luminance and chrominance) of the compensated 
right Image output by the disparity compensator 400 to output luminance 
residual texture and chrominance residual texture, and the auxiliary 
component encoders 340 and 360 encode the luminance residual texture 
and the chrominance residual texture to output quantized luminance 
residual texture and quantized chrominance residual texture. 

The variable length encoder 600 performs variable length 
encoding on the quantized video object and the motion vector output by 
the video object encoder 200, the quantized horizontal disparity map 
output by the auxiliary component encoder 320, the quantized luminance 
residual texture output by the auxiliary component encoder 340, and the 
quantized chrominance residual texture output by the auxiliary component 
encoder 360, and outputs an encoding stream. 

FIG. 11 shows an encoding stream output by the encoder 
according to the third preferred embodiment of the present Invention. As 
shown, the encoding stream output by the encoder Includes macro blocks 
of the encoded video object, macro blocks of the auxiliary component 
AC[0] for the encoded horizontal disparity map, macro blocks of the 
auxiliary component AC[1] for the encoded luminance residual texture, 
and macro blocks of the auxiliary component AC[2] for the encoded 
chrominance residual texture. 

FIG. 10 shows a decoder according to the third preferred 
embodiment of the present Invention. 
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As shown, the decoder comprises a variable length decoder 700, 
a video object decoder 800, auxiliary component decoders 920, 940, and 
960, a disparity compensator 1000, and adders 1100 and 1200. 

The components of FIG. 10 which perfomn the same or similar 
functions as those of the decoder of FIG, 7 have the same reference 
numerals, and no repeated descriptions on the components of FIG. 10 
which execute the same operations as those of the components of FIG. 7 
will be provided. 

The variable length decoder 700 performs variable length 
decoding on the encoding stream output by the encoder and shown in FIG. 
11, and outputs quantized data of the video object, a motion vector, 
quantized data of the horizontal disparity map, quantized data of the 
luminance residual texture, and quantized data of the chrominance 
residual texture. 

The auxiliary component decoder 940 receives the quantized 
data of the luminance residual texture and the motion vector, and decodes 
the luminance residual texture. 

The auxiliary component decoder 960 receives the quantized 
data of the chrominance residual texture and the motion vector, and 
decodes the chrominance residual texture. 

The disparity compensator 1000 receives a reconstructed left 
image and a reconstoicted horizontal disparity map," and performs 
disparity compensation based on a disparity vector of the horizontal 
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dispanty map. 

The adder 1100 adds the disparity-compensated luminance 
texture from among the data disparity-compensated by the disparity 
compensator 1000 and the reconstructed luminance residual texture 
5 output by the auxiliary component decoder 940 to reconstruct the 
luminance component of the right image. 

The adder 1200 adds the disparity-compensated chrominance 
texture from among the disparity-compensated data and the reconstructed 
chrominance residual texture output by the auxiliary component decoder 
1 0 940 to reconstruct the chrominance component of the right image. 

FIGs. 12 and 13 respectively show an encoder and a decoder 
according to a fourth preferred embodiment of the present Invention. The 
encoder and the decoder establishes the 
video_objectJayer_shape_extenslon which has three auxiliary 
components of the horizontal disparity type, the vertical disparity type, and 
the luminance residual texture type which are additionally defined with 
reference to FIG. 4, and perfonns encoding and decoding. 

That is, the pixel-based horizontal disparity map found with 
reference to the left Image Is allocated to aux_comp_type[0], the pixel- 
based vertical disparity map found with reference to the left Image is 
allocated to aux_comp_typeI1]. and the luminance residual texture which 
is residual image data on the luminance component Is allocated to 
aux_comp_type[2]. The encoder and the decoder are usable for simple 
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images which do not greatly influence deterioration of Image quality when 
perfonriing encoding without chrominance residual texture information 
which is residual Image data on the chrominance component, or for 
images which require no precise reconstruction. 

As shown in FIG. 12, the encoder according to the fourth 
preferred embodiment of the present invention comprises a disparity 
estimator 100, a video object encoder 200, auxiliary component encoders 
320, 340, and 380, a disparity compensator 420, an adder 500, and a 
variable length encoder 600. 

The components of FIG. 12 which perform the same or similar 
functions as those of the encoder of FIG. 6 have the same reference 
numerals, and no repeated descriptions on the components of FIG. 12 
which execute the same operations as those of the components of FIG. 6 
will be provided. 

The disparity estimator 100 receives left and right images, and 
finds a pixel-based horizontal disparity map and a vertical disparity map of 
the right image with reference to the left image. 

The auxiliary component encoder 380 receives the vertical 
disparity map output by the disparity estimator 100 and the motion vector 
output by the video object encoder 200, generates a quantized vertical 
disparity map, outputs the quantized vertical disparity map, and outputs a 
vertical disparity map obtained by reconstmcting the quantized vertical 
disparity map. 
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The disparity compensator 420 performs disparity compensation 
based on pixel values (luminance) of the reconstructed left image output 
by the video object encoder 200, the horizontal disparity vectore of the 
reconstructed horizontal disparity map output by the auxiliary component 
encoder 320, and the vertical disparity vectors of the reconstructed vertical 
disparity map output by the auxiliary component encoder 380, and outputs 
pixel values (luminance) of the compensated right image. 

The adder 500 performs a residual process on the pixel value 
(luminance) of the right image and the pixel value (luminance) of the 
disparity-compensated right image output by the disparity compensator 
420 to output luminance residual texture, and the auxiliary component 
encoder 340 encodes the luminance residual texture to output quantized 
luminance residual texture. 

The variable length encoder 600 performs variable length 
encoding on the quantized video object and the motion vector output by 
the video object encoder 200, the quantized horizontal disparity map 
output by the auxiliary component encoder 320. the quantized vertical 
disparity map output by the auxiliary component encoder 380, and the 
quantized luminance residual texture output by the auxiliary component 
encoder 360, and outputs an encoding stream. 

FIG. 14 shows an encoding stream output by the encoder 
according to the fourth preferred embodiment of the present Invention. As 
shown, the encoding stream output by the encoder includes macro blocks 
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Of the encoded video object, macro blocks of the auxiliary component 
AC[01 for the encoded horizontal disparity map. macro blocks of the 
auxiliary component AC[1] for the encoded vertical disparity map. and 
macro blocks of the auxiliary component AC[2] for encoded luminance 
residual texture. 

FIG. 13 shows a decoder according to the fourth preferred 
embodiment of the present invention. 

As shown, the decoder comprises a variable length decoder 700, 
a video object decoder 800, auxiliary component decoders 920, 940, and 
980, a disparity compensator 2000, and an adder 1 100. 

The components of FIG. 13 which perfonn the same or similar 
functions as those of the decoder of FIG. 7 have the same reference 
numerals, and no repeated descriptions on the components of FIG, 13 
which execute the same operations as those of the components of FIG. 7 
will be provided. 

The variable length decoder 700 perfonns variable length 
decoding on the encoding stream output by the encoder as shown in FIG. 
14, and outputs quantized data of the video object, a motion vector, 
quantized data of the horizontal disparity map, quantized data of the 
vertical disparity map, and quantized data of the luminance residual 
texture. 

The auxiliary component decoder 980 receives the quantized 
data of the vertical disparity map and the motion vector, and decodes the 
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vertical disparity map. 

The disparity compensator 2000 receives a reconstructed left 
image, a reconstructed horizontal disparity map, and a reconstructed 
vertical disparity map, and perfonns disparity compensation based on 
disparity vectors of the horizontal disparity map and the vertical disparity 
map. 

The adder 1100 adds the disparity-compensated luminance 
texture from among the data disparity-compensated by the disparity 
compensator 2000 and the reconstructed luminance residual texture 
output by the auxiliary component decoder 940 to reconstruct the 
luminance component of the right image. 

FIGs. 15 and 16 respectively show an encoder and a decoder 
according to a fifth preferred embodiment of the present invention. The 
encoder and the decoder establish the 

video_object_layer_shape_extension which has four auxiliary components 
of the horizontal disparity type, the vertical disparity type, the luminance 
residual texture type, and the chrominance residual texture type which are 
additionally defined when the number of auxiliary components of the MAC 
is increased to equal to or more than four as shown in FIG. 5, and 
performs encoding and decoding. 

That is, the encoder and the decoder allocates the pixel-based 
horizontal disparity map found with reference to the left image to 
aux_comp_typeI01, the pixel-based vertical disparity map found with 
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reference to the left image to aux_comp_typeI1l. the luminance residual 
texture which is residual image data on the luminance component to 
aux_comp_type[2], and the chrominance residual texture which is residual 
image data on the chrominance component to aux__comp_type[3] to 
perform encoding and decoding. 

The above-described fifth preferred embodiment can be 
applicable to the case of reconstructing high-quality images by using ail 
kinds of auxiliaiy information on the right image. 

As shown in FIG. 15, the encoder according to the fifth preferred 
embodiment of the present invention comprises a disparity estimator 100, 
a video object encoder 200, auxiliary component encoders 320, 340, 360, 
and 380, a disparity compensator 420, an adder 500, and a variable 
length encoder 600. 

The components of FIG. 15 which perform the same or similar 
functions as those of the encoder of FIG. 12 have the same reference 
numerals, and no repeated descriptions on the components of FIG. 15 
which execute the same operations as those of the components of FIG. 
12 will be provided. 

The disparity estimator 100 receives left and right images, and 
finds a pixel-based horizontal disparity map and a vertical disparity map of 
the right image with reference to the left image. 

The auxiliary component encoder 380 receives the vertical 
disparity map output by the disparity estimator 100 and the motion vector 
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output by the video object encoder 200, generates a quantized vertical 
disparity map, outputs the quantized vertical disparity map, and outputs a 
vertical disparity map obtained by reconstructing the quantized vertical 
disparity map. 

The disparity compensator 420 performs disparity compensation 
based on pixel values (luminance and chrominance) of the reconstructed 
left image output by the video object encoder 200, the horizontal disparity 
vectors of the reconstructed horizontal disparity map output by the 
auxiliary component encoder 320, and the vertical disparity vectors of the 
reconstructed vertical disparity map output by the auxiliary component 
encoder 380, and outputs pixel values (luminance and chrominance) of 
the compensated right image. 

The adder 500 performs a residual process on the pixel values 
(luminance and chrominance) of the right image and the pixel values 
(luminance and chrominance) of the disparity-compensated right image 
output by the disparity compensator 420 to output luminance residual 
texture and chrominance residual texture, and the auxiliary component 
encoder 340 and the auxiliary component encoder 360 respectively 
encode the luminance residual texture and the chrominance residual 
texture to output quantized luminance residual texture and quantized 
chrominance residual texture. 

The variable length encoder 600 perfonns variable length 
encoding on the quantized video object and the motion vector output by 
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the Video object encoder 200, the quantized horizontal disparity map 
output by the auxiliary component encoder 320, the quantized vertical 
disparity map output by the auxiliary component encoder 380, the 
quantized luminance residual texture output by the auxiliary component 
encoder 340, and the quantized chrominance residual texture output by 
the auxiliary component encoder 360, and outputs an encoding stream. 

FIG. 17 shows an encoding stream output by the encoder 
according to the fifth preferred embodiment of the present Invention. As 
shown, the encoding stream output by the encoder includes macro blocks 
of the encoded video object, macro blocks of the auxiliary component 
AC[0] for the encoded horizontal disparity map, macro blocks of the 
auxiliary component AC[1] for the encoded vertical disparity map, macro 
blocks of the auxiliary component AC[2] for the encoded luminance 
residual texture, and macro blocks of the auxiliary component ACI3] for 
the encoded chrominance residual texture. 

FIG. 16 shows a decoder according to the fifth preferred 
embodiment of the present invention. 

As shown, the decoder comprises a variable length decoder 700, 
a video object decoder 800, auxiliary component decoders 920, 940, 960, 
and 980, a disparity compensator 2000, and adders 1 100 and 1200. 

The components of FIG. 16 which perform the same or similar 
functions as those of the decoder of FIG. 13 have the same reference 
numerals, and no repeated descriptions on the components of FIG. 16 
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Which execute the same operations as those of the components of FIG. 
13 will be provided. 

The variable length decoder 700 perfomis variable length 
decoding on the encoding stream output by the encoder and shown in FIG. 
17, and outputs quantized data of the video object, a motion vector, 
quantized data of the horizontal disparity map, quantized data of the 
vertical disparity map, quantized data of the luminance residual texture, 
and quantized data of the chrominance residual texture. 

The disparity compensator 2000 receives a reconstructed left 
image, a reconstructed horizontal disparity map, and a reconstructed 
vertical disparity map, and perfonns disparity compensation based on 
disparity vectors of the horizontal disparity map and the vertical .disparity 
map. 

The adder 1100 adds the disparity-compensated luminance 
texture from among the data disparity-compensated by the disparity 
compensator 2000 and the reconstructed luminance residual texture 
output by the auxiliary component decoder 940 to reconstruct the 
luminance component of the right image. 

The adder 1200 adds the disparity-compensated chrominance 
texture from among the data disparity-compensated by the disparity 
compensator 2000 and the reconstructed chrominance residual texture 
output by the auxiliary component decoder 960 to reconstruct the 
chrominance component of the right image. 
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The video object encoders 200, the auxiliary component encoders 
320. 340. 360, and 380. the video object decoder 800. and the auxiliary 
component decoders 920. 940. 960. and 980 will now be described in 
further detail. 

5 FIG. 18 shows a detailed block diagram for the video object 

encoder 200 according to the preferred embodiments of the present 
invention. 

As shown, the video object encoder 200 comprises an encoding 
unit 220. a decoding unit 240. a motion estimator 260, and a motion 
10 compensator 280. 

The encoding unit 220 performs a discrete cosine transform on 
the video object data (left image) and the motion-compensated data 
having undergone the residual process, quantizes the discrete-cosine- 
transformed data, and outputs the quantized video object. The encoding 
unit 220 comprises an adder 221 for performing a residual process on the 
video object data and the motion-compensated data, a DCT (discrete 
cosine transformer) 222 for performing a discrete cosine transform on the 
residual data output by the adder 221. and a quantizer 223 for quantizing 
the data output by the DCT 222. 

The decoding unit 240 dequantizes the quantized video object 
output by the encoding unit 220. and perfonns an inverse discrete cosine 
transfomi on the de-quantized data to reconstruct the video object data. 
The decoding unit 240 comprises a dequantizer 241 for dequantizing the 
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quantized video object output by the encoding unit 220, an IDCT (inverse 
discrete cosine transformer) 242 for performing an inverse discrete cosine 
transfomi on tlie data output by the dequantizer 241, an adder 243 for 
adding the video object output by the IDCT 242 and the mbtion- 
5 compensated data to reconstruct the video object data, and a memory 
244 for storing the reconstructed left Image output by the adder 243. 

The motion estimator 260 compares the video object data (left 
image) with the reconstructed left Image of a previous frame stored in the 
memory 244 to output a motion vector MV. 
10 The motion compensator 280 compares the motion vector output 

by the motion estimator 260 with the left Image of a previous frame stored 
in the memory 244 to output motion compensation data. 

FIG. 19 shows a detailed block diagram for the auxiliary 
component encoder 300 according to the preferred embodiments of the 
15 present invention. 

As shown, the auxiliary component encoder 300 comprises an 
encoding unit 310, a decoding unit 330, and a motion compensator 350. 

The encoding unit 310 performs a discrete cosine transform on 
the auxiliary component data and the motion-compensated data having 
20 undergone the residual process, quantizes the dlscrete-coslne- 
transformed data, and outputs the quantized auxiliary component data. 
The encoding unit 310 comprises an adder 311 for performing a residual 
process on the auxiliary component data and the motion-compensated 
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data, a DCT (discrete cosine transformer) 312 for performing a discrete 
cosine transform on the residual data output by the adder 311, and a 
quantizer 313 for quantizing the data output by the DCT 312. 

The decoding unit 330 dequantizes the quantized auxiliary 
5 component data output by the encoding unit 310, and performs an inverse 
discrete cosine transform on the de-quantized data to reconstruct the 
auxiliary component data. The decoding unit 330 comprises a dequantizer 
331 for dequantizing the quantized auxiliary component data output by the 
encoding unit 310, an IDCT (inverse discrete cosine transformer) 332 for 

10 performing an inverse discrete cosine transform on the data output by the 
dequantizer 331, an adder 333 for adding the auxiliary component data 
output by the IDCT 332 and the motion-compensated data to reponstruct 
the auxiliary component data, and a memory 334 for storing the 
reconstructed auxiliary component data output by the adder 333. 

15 The motion compensator 350 compares the motion vector output 

by the motion estimator 260 of the video object encoder 200 with the 
auxiliary component data of a previous frame stored in the memory 334 to 
output motion compensation data on the auxiliary component. 

FIG. 20 shows a detailed block diagram for the video object 

20 decoder 800 according to the preferred embodiments of the present 
invention. 

As shown, the video object decoder 800 comprises a dequantizer 
810, an IDCT 820. an adder 830, a motion compensator 850, and a 
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memory 840. 

The dequantlzer 810 dequantlzes the quantized data of the video 
object output by the variable length decoder, the IDCT 820 performs an 
inverse discrete cosine transform on the data output by the dequantlzer 
810, and the motion compensator 850 compares the reconstructed video 
object data of a previous frame stored In the memory 840 with the motion 
vector to compensate for motion, and outputs compensated motion vector 
data. 

The adder 830 adds the video object output by the IDCT 820 and 
the motion-compensated vector data output by the motion compe/isator 
850 to output reconstructed video object data. 

FIG. 21 shows a detailed block diagram for the auxiliary 
component decoder 900 according to the preferred embodiments of the 
present invention. 

As shown, the auxiliary component decoder 900 comprises a 
dequantlzer 901. an IDCT 902, an adder 903, a motion compensator 905, 
and a memory 904. 

The dequantlzer 901 dequantlzes the quantized data of the 
auxiliary component output by the variable length decoder, the IDCT 902 
performs an Inverse discrete cosine transfomn on the data output by the 
dequantlzer 901, and the motion compensator 905 compares the 
reconstaicted auxiliary component data of a previous frame stored In the 
memory 904 with the motion vector to compensate for motion, and outputs 
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compensated motion vector data. 

The adder 903 adds the auxiliary component data output by the 
IDCT 902 and the motion-compensated vector data output by the motion 
compensator 905 to output reconstructed video object data. 

As described, compatibility for using the current IVIPEG-4 
encoding techniques and systems is provided since the stereoscopic 
video are encoded by using the MAC of the IVlPEG-4 standard according 
to the preferred embodiments of the present invention. 

Also, synchronization of between the right and left images is 
simply perfonned since the encoding streams of the right and left images 
are output as a single encoding stream. 

Further, encoding efficiencies are improved by allocating four 
components of auxiliary infonnation of the right image to the MAC in 
various ways, selecting thereof according to image quality levels desired 
by a user or a producer, and encoding the same. • 

While this invention has been described in connection with what is 
presently considered to be the most practical and preferred embodiment, 
it is to be understood that the invention is not limited to the disclosed 
embodiments, but. on the contrary, is intended to cover various 
modifications and equivalent arrangements included within the spirit and 
scope of the appended claims. 

For example, the auxiliary infonnatioh is allocated using the MAC 
defined by the MPEG-4 standard, and in addition, information defined by 
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Other protocols can be used. 

As described, compatibility for using the existing MPEG-4 
encoding techniques and systems is maintained, and complexity of 
synchronization of between the right and left images is minimized. 

In addition, encoding efficiency is Improved by selectively 
controlling the image quality and encoding it according to importance and 
complexity of the images. 



